{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Unknown Dataset Sizes\n",
    "\n",
    "This notebook demonstrates the features built into OpenDP to handle unknown or private dataset sizes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load exemplar dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-15T16:41:40.471981Z",
     "iopub.status.busy": "2021-04-15T16:41:40.458021Z",
     "iopub.status.idle": "2021-04-15T16:41:40.634785Z",
     "shell.execute_reply": "2021-04-15T16:41:40.635247Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "data_path = os.path.join('.', 'data', 'PUMS_california_demographics_1000', 'data.csv')\n",
    "with open(data_path) as data_file:\n",
    "    data = data_file.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the private data, we see this dataset has 1000 observations (rows).\n",
    "Oftentimes the number of observations is public information.\n",
    "For example, a researcher might run a random poll of 1000 respondents and publicly announce the sample size.\n",
    "\n",
    "However, there are cases where simply the number of observations itself can leak private information.\n",
    "For example, if a dataset contained all the individuals with a rare disease in a community,\n",
    "then knowing the size of the dataset would reveal how many people in the community had that condition.\n",
    "In general, any given dataset may be some well-defined subset of a population.\n",
    "The given dataset's size is equivalent to a count query on that subset,\n",
    "so we should protect the dataset size just as we would protect any other query we want to provide privacy guarantees for.\n",
    "\n",
    "OpenDP assumes the sample size is private information.\n",
    "If you know the dataset size (or any other parameter) is publicly available,\n",
    "then you are free to make use of such information while building your measurement.\n",
    "\n",
    "OpenDP will not assume you truthfully or correctly know the size of the dataset.\n",
    "Moreover, OpenDP cannot respond with an error message if you get the size incorrect;\n",
    "doing so would permit an attack whereby an analyst could repeatedly guess different dataset sizes until the error message went away,\n",
    "thereby leaking the exact dataset size.\n",
    "\n",
    "If we know the dataset size, we can incorporate it into the analysis as below,\n",
    "where we provide `size` as an argument to the release of a sum on age.\n",
    "While the \"sum of ages\" is not a particularly useful statistic, it's plenty capable of demonstrating the concept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-15T16:41:40.645482Z",
     "iopub.status.busy": "2021-04-15T16:41:40.644181Z",
     "iopub.status.idle": "2021-04-15T16:41:40.686094Z",
     "shell.execute_reply": "2021-04-15T16:41:40.685529Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DP sum: 44798\n"
     ]
    }
   ],
   "source": [
    "from opendp.trans import *\n",
    "from opendp.meas import make_base_geometric\n",
    "from opendp.mod import enable_features\n",
    "enable_features(\"contrib\")\n",
    "\n",
    "# Define parameters up-front\n",
    "# Each parameter is either a guess, a DP release, or public information\n",
    "var_names = [\"age\", \"sex\", \"educ\", \"race\", \"income\", \"married\"]  # public information\n",
    "size = 1000 # public information\n",
    "age_bounds = (0, 100) # an educated guess\n",
    "constant = 38 # average age for entire US population (public information)\n",
    "\n",
    "dp_sum = (\n",
    "    # Load data into a dataframe of string columns\n",
    "    make_split_dataframe(separator=\",\", col_names=var_names) >>\n",
    "    # Selects a column of df, Vec<str>\n",
    "    make_select_column(key=\"age\", TOA=str) >>\n",
    "    # Cast the column as Vec<Int>\n",
    "    make_cast(TIA=str, TOA=int) >>\n",
    "    # Impute missing values to 0\n",
    "    make_impute_constant(constant) >>\n",
    "    # Clamp age values\n",
    "    make_clamp(bounds=age_bounds) >>\n",
    "    # Resize with the known `size`\n",
    "    make_bounded_resize(size=size, bounds=age_bounds, constant=constant) >>\n",
    "    # Aggregate\n",
    "    make_sized_bounded_sum(size=size, bounds=age_bounds) >>\n",
    "    # Noise\n",
    "    make_base_geometric(scale=1.)\n",
    ")\n",
    "\n",
    "release = dp_sum(data)\n",
    "print(\"DP sum:\", release)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Providing incorrect dataset size values\n",
    "\n",
    "However, if we provide an incorrect value of `n` we still receive an answer.\n",
    "\n",
    "`make_sum_measurement` is just a convenience constructor for building a sum measurement from a `size` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-15T16:41:40.694235Z",
     "iopub.status.busy": "2021-04-15T16:41:40.693539Z",
     "iopub.status.idle": "2021-04-15T16:41:40.711013Z",
     "shell.execute_reply": "2021-04-15T16:41:40.711551Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DP sum (n=200):  8809\n",
      "DP sum (n=1000): 44796\n",
      "DP sum (n=2000): 82799\n"
     ]
    }
   ],
   "source": [
    "preprocessor = (\n",
    "    make_split_dataframe(separator=\",\", col_names=var_names) >>\n",
    "    make_select_column(key=\"age\", TOA=str) >>\n",
    "    make_cast_default(TIA=str, TOA=int) >>\n",
    "    make_clamp(age_bounds)\n",
    ")\n",
    "\n",
    "def make_sum_measurement(size):\n",
    "    return make_bounded_resize(size=size, bounds=age_bounds, constant=constant) >> \\\n",
    "           make_sized_bounded_sum(size=size, bounds=age_bounds) >> \\\n",
    "           make_base_geometric(scale=1.0)\n",
    "\n",
    "lower_n = (preprocessor >> make_sum_measurement(size=200))(data)\n",
    "real_n = (preprocessor >> make_sum_measurement(size=1000))(data)\n",
    "higher_n = (preprocessor >> make_sum_measurement(size=2000))(data)\n",
    "\n",
    "print(\"DP sum (n=200):  {0}\".format(lower_n))\n",
    "print(\"DP sum (n=1000): {0}\".format(real_n))\n",
    "print(\"DP sum (n=2000): {0}\".format(higher_n))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis with no provided dataset size\n",
    "If we do not believe we have an accurate estimate for `size` we can instead pay some of our privacy budget\n",
    "to estimate the dataset size.\n",
    "Then we can use that estimate in the rest of the analysis.\n",
    "Here is an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-04-15T16:41:40.731918Z",
     "iopub.status.busy": "2021-04-15T16:41:40.731318Z",
     "iopub.status.idle": "2021-04-15T16:41:40.740106Z",
     "shell.execute_reply": "2021-04-15T16:41:40.739600Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DP count: 1000\n",
      "DP sum: 44795\n"
     ]
    }
   ],
   "source": [
    "# First, make the measurement\n",
    "dp_count = (\n",
    "    make_split_dataframe(separator=\",\", col_names=var_names) >>\n",
    "    make_select_column(key=\"age\", TOA=str) >>\n",
    "    make_count(TIA=str) >>\n",
    "    make_base_geometric(scale=1.)\n",
    ")\n",
    "dp_count_release = dp_count(data)\n",
    "print(\"DP count: {0}\".format(dp_count_release))\n",
    "\n",
    "dp_sum = preprocessor >> make_sum_measurement(dp_count_release)\n",
    "dp_sum_release = dp_sum(data)\n",
    "print(\"DP sum: {0}\".format(dp_sum_release))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that our privacy usage has increased because we apportioned some epsilon for both the release count of the dataset,\n",
    "and the mean of the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenDP `resize` vs. other approaches\n",
    "The standard formula for the mean of a variable is:\n",
    "$\\bar{x} = \\frac{\\sum{x}}{n}$\n",
    "\n",
    "The conventional, and simpler, approach in the differential privacy literature, is to: \n",
    "\n",
    "1. compute a DP sum of the variable for the numerator\n",
    "2. compute a DP count of the dataset rows for the denominator\n",
    "3. take their ratio\n",
    "\n",
    "This is sometimes called a 'plug-in' approach, as we are plugging-in differentially private answers for each of the\n",
    "terms in the original formula, without any additional modifications, and using the resulting answer as our\n",
    "estimate while ignoring the noise processes of differential privacy. While this 'plug-in' approach does result in a\n",
    "differentially private value, the utility here is generally lower than the solution in OpenDP.  Because the number of\n",
    "terms summed in the numerator does not agree with the value in the denominator, the variance is increased and the\n",
    "resulting distribution becomes both biased and asymmetrical, which is visually noticeable in smaller samples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We have noticed that for the same privacy loss,\n",
    "the distribution of answers from OpenDP's resizing approach to the mean is tighter around the true dataset value (thus lower in error) than the conventional plug-in approach.\n",
    "\n",
    "*Note, in these simulations, we've shown equal division of the epsilon for all constituent releases,\n",
    "but higher utility (lower error) can be generally gained by moving more of the epsilon into the sum,\n",
    "and using less in the count of the dataset rows, as in earlier examples.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}